1. Technical Field
The invention has application in the field of oculometer design. Oculometers may be used to measure the eye gaze direction, as well as the fixation duration and dual eye binocular convergence point. Such oculometers have many potential applications in the medical, scientific, engineering, manufacturing, military, and entertainment domains. Example applications include use of an oculometer as a tool for the medical diagnosis of ocular functions, as an aid to the paraplegic handicapped, for the measurement of ocular functions and workload in human factors studies, as a measure of subject training, as a tool for fatigue monitoring, as part of an electronic safety net to detect performance degradation due to pilot incapacitation in piloted and tele-operated vehicles, as a component of an electronic intelligent pilot-vehicle interface used for adaptive aiding in piloted and tele-operated vehicles, for task scan analysis including measuring situation awareness, for human operator control of machines and interaction with computer games, and for advertisement and usability analysis. Oculometers may be designed for use with head-mounted video displays such as those that have been developed for virtual reality, stereographic displays, monocular or binocular vision helmet-mounted displays, and night vision goggles. These displays are used in piloted helicopters, vehicles, and control stations for teleoperated robotics.
Oculometers may be used as an eyetracker to control computerized machines from an electronic video display by the ocular gaze point of regard and fixation duration. Examples of machine control by ocular functions are: (1) updating computer generated information displays, (2) selecting panel switches and instruments, (3) controlling the fidelity of computer generated imagery scene inserts in simulations, (4) controlling the viewing direction of remotely located cameras, (5) controlling the movement of teleoperated robotics platforms or vehicles, (6) selecting display subareas for automated scene analysis in aided target recognition, (7) designating targets from direct sight or from a sensor display, and (8) weapon system pointing.
Oculometers may have particular applications to time shared concurrent tasks where the hands are involved in a continual time critical pilotage task and the eyes may be used intermittently to control a discrete task. The use of this invention enables both tasks to share a common visual working area with overlaid visual images. In this way, task interference is reduced by dedicating eye-movements and visual attention to the same working surface. An example of such an application would be single pilot nap-of-earth low-level helicopter flight while updating onboard heads-up displays. A similar application is teleoperations of remote vehicles from video displays with camera control. Another such application is to the operation of completely enclosed armored vehicles with “transparent” or “see through” armor where the operator sees a video projection of the outside scene as recorded by externally mounted cameras and relayed to internal monitors; the operator would use the invention to control displays overlaid on the scene projection while concurrently performing the vehicle pilotage task. Similar comments apply to the piloting of “glass cockpit” designs for completely enclosed high performance aircraft.
2. Description of the Related Art
A common technology for oculometers (i.e., eye-trackers) is videooculography based upon the optical measurement of reflected light from the human eye, commonly near-infrared light for an image of the pupil. In its simplest form, an oculometer contains a single infrared light source which is directed at the eye and the reflected light is imaged onto a charge-injection (CID) or charge-coupled device (CCD) sensor array. The image of the eye is then electronically processed to determine the corneal reflection, the pupil centroid orientation, or both. These parameters are used to determine the angular location of the eye relative to the camera within a fair degree of accuracy. The technology is either head-mounted or mounted in a panel in front of the user.
Many head-mounted oculometers typically comprise a light source that illuminates the eye to be tracked, and a single light sensor that captures rays of light that are reflected from the eye. Although such oculometers provide an indication of eye position and, therefore, gaze direction, the use of a single light sensor presents various potential limitations or drawbacks. For example, a single sensor may not receive the rays reflected off of the cornea or eye interior in cases in which the user's gaze is fixed upon an object positioned at an extreme angle relative to the forward-looking direction (e.g., when the wearer is gazing laterally).
The optics for the panel mounted system are typically mounted in front of the user and directed toward his or her face. The panel mounted system is limited to low ambient light levels and objects that the user may need to work with cannot be readily placed between the face and the optics. For a single sensor system, a servomechanism is used to keep the optics aligned on the user's eye by tracking the image of the eye-orbit in the face, and the servomechanism adjustment is noticeable to users following a head movement. Excessive head movements and interference of the optical path by facial features (such as the user's nose) are not tolerated. More recent developments use multiple high definition camera systems that mounted about the workspace of the user track the face and eye with visual light as well as infrared; the eye is located from the iris or limbus image relative to the face position in the workspace. The determination is commonly made from the best camera view among multiple views.
The oculometer determines the angular location of the eye relative to the sensor array within a fair degree of accuracy. The measurement of head position and orientation for the head-mounted system allows the determination of eye position and orientation in the workspace, and therefore computation of the eye-point of regard. Similarly, determination of the range from the sensor to the eye by either say, ultrasonic or automatic image focusing, enables the computation of eye-point of regard for the panel system; a further method is image sizing of known features such as markers placed on the face or of the face itself in images with the eye orbit including such as the limbus of the eye.
The accuracy of the technology is roughly about +/− one degree in practice and is limited by the processing of the pupil image from which the image centroid and orientation are determined; these parameters are used to estimate the angular location of the eye relative to the camera. In earlier designs, the gaze direction is determined from the glint offset in the camera image alone for a single source placed to the side; the light source is directed to the side of the cornea where the shape is more cylindrical and the glint shifts with eye rotation. In some designs using near infrared, the line-of sight is measured from the offset of the pupil image centriod from the corneal surface glint for a light source collinear with the camera or that for several sources placed about the camera. Because of the nearly spherical shape of the corneal in the region about the visual axis, the glint from a light source directed to that area is fixed in position on the corneal image independent of eye rotation. The pupil image is bright for a collinear source because of the retinal reflection, but dark for a light source positioned to the side. For multiple camera systems mounted about the workspace using visible light, a common method is the offset of the limbus centroid within the eye orbit of the face. Essentially, the technology is based on the offset of a centroid for an ellipsoid in the eye-image from the location of a known feature either a glint point or the face. A more recent development is the location of eye-point of regard by triangulation of glint points in the pupil image for corneal surface reflections (i.e., glint) of light sources that are located in the workspace at known positions.
The technology is most accurate for a front camera view of the eye; however, the accuracy decreases with offset up to 45-degrees. This because much of the present technology is based on pupil (or limbus) image processing using simple perspective computations. However, a problem is the image distortion caused by corneal surface refraction. The accuracy decreases with side-view because of the non-uniform distortions in the pupil image caused by the corneal surface refraction with pupil offset. This non-linearity in distortion causes the centroid computed for the image ellipsoid to be displaced from the centroid for the true image resulting in computational error, a problem the present technology does not correct. Furthermore, the technology estimates only the principal axis of sight and does not account for the effect on sight direction of the torsional roll of the eye with accompanying vertical vergence eye movement that is induced during the tilting of the head a common occurrence during target tracking and user motion. In this application, we disclose stereo-image eye-tracker designs that incorporate novel image processing techniques particular to stereo images including that of pupil image reconstruction, that provide the advantage of tracking with increased accuracy over a wider field of view.